Papers with contrastive learning objective
Capturing the Relationship Between Sentence Triplets for LLM and Human-Generated Texts to Enhance Sentence Embeddings (2024.findings-eacl)
Copied to clipboard
| Challenge: | Recent advances in building sentence embedding models have centered on replacing traditional human-generated text datasets with those generated by LLMs. |
| Approach: | They propose a loss function that incorporates Positive-Negative sample Augmentation within the contrastive learning objective to enhance sentence embeddings using both human and LLM-generated datasets. |
| Outcome: | The proposed model mitigates the sentence anisotropy problem in Wikipedia corpus and improves Spearman’s correlation in standard Semantic Textual Similarity (STS) tasks (+1.47% compared to CLHAIF). |
RoCBert: Robust Chinese Bert with Multimodal Contrastive Pretraining (2022.acl-long)
Copied to clipboard
| Challenge: | Large-scale pretrained language models have achieved SOTA results on NLP tasks but are vulnerable to adversarial attacks especially for logographic languages like Chinese. |
| Approach: | They propose a pretrained Chinese Bert that is robust to various forms of adversarial attacks like word perturbation, synonyms, typos, etc. |
| Outcome: | The proposed model outperforms baselines on 5 Chinese NLU tasks without sacrificing performance on clean testsets. |
Fine-grained Contrastive Learning for Relation Extraction (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods assume all silver labels are accurate and treat them equally, but distant supervision is noisy–some silver labels more reliable than others. |
| Approach: | They propose a noise-aware contrastive learning approach that leverages fine-grained information about which silver labels are and are not noisy to improve the quality of learned relationship representations. |
| Outcome: | The proposed approach improves relation extraction performance over state-of-the-art methods on several RE benchmarks. |
Learning the Visualness of Text Using Large Vision-Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Visual text evokes an image in a person’s mind, while non-visual text fails to do so. |
| Approach: | They propose a method to automatically detect visualness in text to enable text-to-image retrieval and generation models to augment text with relevant images. |
| Outcome: | The proposed method performs better than several baseline models and heuristics for the task. |
Generalizing Conversational Dense Retrieval via LLM-Cognition Data Augmentation (2024.acl-long)
Copied to clipboard
| Challenge: | Existing conversational dense retrieval models view a conversation as a fixed sequence of questions and responses, and these alternate conversations are unrecorded. |
| Approach: | They propose a framework for generalizing Conversational dense retrieval via LLM-cognition data Augmentation (ConvAug) they first generate multi-level augmented conversations to capture the diverse nature of conversational contexts. |
| Outcome: | The proposed framework generalizes Conversational dense retrieval via LLM-cognition data Augmentation on four public datasets. |
CKDST: Comprehensively and Effectively Distill Knowledge from Machine Translation to End-to-End Speech Translation (2023.findings-acl)
Copied to clipboard
| Challenge: | End-to-end speech-totext translation (ST) data are limited due to the limited resources. |
| Approach: | They propose a knowledge distillation framework for speech translation that integrates knowledge from machine translation and decouples knowledge from non-target class knowledge. |
| Outcome: | The proposed framework outperforms state-of-the-art models on a benchmark dataset. |
Self-Guided Contrastive Learning for BERT Sentence Representations (2021.acl-long)
Copied to clipboard
| Challenge: | Existing methods to derive sentence embeddings from pre-trained Transformers are unclear . a self-guided training method is used to fine-tune BERT in a supervised fashion . |
| Approach: | They propose a contrastive learning method that utilizes self-guidance to improve BERT sentence representations. |
| Outcome: | The proposed method is more effective than baselines on diverse sentence-related tasks and robust to domain shifts. |
Improving Word Translation via Two-Stage Contrastive Learning (2022.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to bilingual lexicon induction (BLI) are limited to two stages, but we propose a robust and effective two-stage contrastive learning framework for the task. |
| Approach: | They propose a two-stage contrastive learning framework for the task . they propose to refine cross-lingual linear maps between static word embeddings via a contrastive objective and integrate it into the self-learning procedure for even more refined cross-linguistic maps. |
| Outcome: | The proposed framework improves cross-lingual maps and word translation capability by integrating it into the self-learning procedure. |
Don’t Judge a Language Model by Its Last Layer: Contrastive Learning with Layer-Wise Attention Pooling (2022.coling-1)
Copied to clipboard
| Challenge: | Recent pre-trained language models (PLMs) have shown competitive performance on many natural language processing tasks. |
| Approach: | They propose a pooling strategy which preserves layer-wise signals captured in each layer and learns digested linguistic features for downstream tasks. |
| Outcome: | The proposed method improves on standard semantic textual similarity and semantic search tasks. |
Unsupervised Sentence Representation Learning with Syntactically Aligned Negative Samples (2025.findings-naacl)
Copied to clipboard
| Challenge: | Existing approaches to sentence representation learning often encounter semantic inconsistencies and feature suppression. |
| Approach: | They propose a method for generating syntactically aligned negative (SAN) samples using a semantic importance-aware Masked Language Model (MLM) approach. |
| Outcome: | The proposed method produces negative samples with substantial textual overlap with the original sentences while conveying different meanings. |
SimCSE: Simple Contrastive Learning of Sentence Embeddings (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for learning universal sentence embeddings are based on unsupervised approaches with only dropout as noise. |
| Approach: | They propose an unsupervised approach that takes an input sentence and predicts itself in a contrastive objective with only standard dropout used as noise. |
| Outcome: | The proposed framework performs on par with previous supervised approaches and can produce superior sentence embeddings from unlabeled or labeled data. |
CoCGAN: Contrastive Learning for Adversarial Category Text Generation (2022.coling-1)
Copied to clipboard
| Challenge: | Experimental results on synthetic and real category text generation datasets demonstrate that CoCGAN can achieve significant improvements over the baseline category text generators. |
| Approach: | They propose to incorporate contrastive learning into adversarial category text generation by using a discriminator to optimize a contrastive learn objective to capture more flexible data-to-class relations and data- to-data relations among training samples. |
| Outcome: | The proposed model improves on synthetic and real category text generation datasets. |
Instances and Labels: Hierarchy-aware Joint Supervised Contrastive Learning for Hierarchical Multi-Label Text Classification (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to hierarchical multi-label text classification (HMTC) ignore the correlation between similar samples and introduce noise . |
| Approach: | They propose a semi-supervised method that uses a label hierarchy to bring text and label embeddings closer to each other by supervised contrastive learning. |
| Outcome: | The proposed method bridges the gap between supervised contrastive learning and HMTC by bringing text and label embeddings closer. |
MABEL: Attenuating Gender Bias using Textual Entailment Data (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for mitigating gender bias in language models are insufficient or inconsistent. |
| Approach: | They propose a method for attenuating gender bias using entailment labels . they use a contrastive learning objective on counterfactually augmented enanglement pairs . |
| Outcome: | The proposed method outperforms previous task-agnostic debiasing approaches on intrinsic and extrinsic metrics and preserves task performance after fine-tuning on downstream tasks. |
Contrastive Learning-based Sentence Encoders Implicitly Weight Informative Words (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Embedding a sentence into a point in a highdimensional continuous space plays a foundational role in the natural language processing. |
| Approach: | They propose to use contrastive loss to fine-tune sentences by inverse word frequency . they also show that more informative words receive greater weight than less informative ones . |
| Outcome: | The proposed method improves the performance of sentence embeddings by weighing them based on information-theoretic quantities. |
Cross-lingual and Multilingual CLIP (2022.lrec-1)
Copied to clipboard
| Challenge: | OpenAI released CLIP, a model that relates the textual and visual domains with unprecedented accuracy. |
| Approach: | They propose to use cross-lingual teacher learning to re-train an English textual encoder using a large dataset of images and captions. |
| Outcome: | The proposed method outperforms baselines on multilingual image-text retrieval while retaining low cost. |
Topic-Guided Sampling For Data-Efficient Multi-Domain Stance Detection (2023.acl-long)
Copied to clipboard
| Challenge: | Stance Detection is a task that aims to identify the attitudes of an author towards a target of interest. |
| Approach: | They propose a topic-guided diversity sampling technique and a contrastive objective to improve stance detection using the produced set. |
| Outcome: | The proposed method outperforms the state-of-the-art on 16 datasets with in-domain and out-of domain evaluations and is more generalizable with an averaged 10.2 F1 on out-domain evaluation. |
SamToNe: Improving Contrastive Loss for Dual Encoder Retrieval Models with Same Tower Negatives (2023.findings-acl)
Copied to clipboard
| Challenge: | Dual encoders have been used for retrieval tasks and representation learning with good results. |
| Approach: | They propose an improved contrastive learning objective by adding queries or documents from the same encoder towers to the negatives. |
| Outcome: | The proposed model improves retrieval quality for both symmetric and asymmetric dual encoders by adding queries or documents from the same encoder towers to the negatives. |
DICE: Data-Efficient Clinical Event Extraction with Generative Models (2023.acl-long)
Copied to clipboard
| Challenge: | EE tasks target specific domains with vague entity boundaries, resulting in a lack of training data. |
| Approach: | They propose a robust and data-efficient generative model for clinical event extraction . they frame event extraction as a conditional generation problem and introduce a contrastive learning objective to decide the boundaries of biomedical mentions. |
| Outcome: | The proposed model is robust and data-efficient for clinical event extraction . it trains an auxiliary mention identification task and event extraction tasks to better identify entity mention boundaries . |
R^3AG: Retriever Routing for Retrieval-Augmented Generation (2026.acl-long)
Copied to clipboard
| Challenge: | Retrieval-augmented generation (RAG) is often bottlenecked by the “one-size-fits-all” retrieval paradigm, as different queries exhibit distinct preferences for different retrievers. |
| Approach: | They propose a novel routing framework that explicitly models the dynamic alignment between queries and retriever capabilities and decomposes retriever capability into two learnable dimensions: retrieval quality and generation utility. |
| Outcome: | Experiments on knowledge-intensive tasks show that R3AG outperforms both the best individual retrievers and state-of-the-art static routing methods. |
Fairness in Automatic Speech Recognition Isn’t a One-Size-Fits-All (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Pre-trained speech models like Whisper exhibit inconsistent group-level performance that varies across domains. |
| Approach: | They fine-tune a Whisper model on the Fair-Speech corpus using basic fine- tuning, demographic rebalancing, gender-swapped data augmentation and a novel contrastive learning objective. |
| Outcome: | The proposed method achieves stable, cross-domain fairness improvements without changes to the training data distribution and with minimal accuracy trade-offs. |
Read As Human: Compressing Context via Parallelizable Close Reading and Skimming (2026.acl-long)
Copied to clipboard
Jiwei Tang, Shilei Liu, Zhicheng Zhang, Qingsong Lv, Runsong Zhao, Tingwei Lu, Langming Liu, Haibin Chen, Yujin Yuan, Hai-Tao Zheng, Wenbo Su, Bo Zheng
| Challenge: | Existing task-aware methods require loading the entire input sequence at once for compression, which suffer from computational inefficiency. |
| Approach: | They propose a framework that adopts an adaptive hybrid reading strategy to reduce computational inefficiency and redundant information in long-context scenarios. |
| Outcome: | Experiments show that RAM outperforms baselines on multiple question answering and summarization benchmarks while delivering up to a 12x speedup on long inputs. |
Mind the Pause: Disfluency-Aware Objective Tuning for Multilingual Speech Correction with LLMs (2026.acl-long)
Copied to clipboard
| Challenge: | Spontaneous speech is rarely fluent, and disfluencies can degrade readability and reliability . a sequence tagger first marks disfluent tokens, and these signals guide instruction fine-tuning . |
| Approach: | They propose a multilingual correction pipeline where a sequence tagger first marks disfluent tokens . they add a contrastive learning objective that penalizes the reproduction of disfluency tokens. |
| Outcome: | The proposed model improves readability and reliability of ASR transcripts in three languages . disfluencies can cause misinterpretations, incoherent responses, poor user experience . |
FIGMA: Towards FIne-Grained Music retrievAl (2026.acl-long)
Copied to clipboard
| Challenge: | Existing music retrieval models fail to retrieve fine-grained musical attributes when using coarse semantic queries. |
| Approach: | They propose a multi-view contrastive architecture that captures high-level semantic context and fine-grained musical attributes within a unified representation space. |
| Outcome: | The proposed method outperforms existing CLAP-based music retrieval models on multiple benchmarks. |